When an image is shot with a camera including an X-Y address scanning type imaging device such as a complementary metal oxides semiconductor (CMOS) imaging device under illumination of a fluorescent lamp, luminance non-uniformity or color non-uniformity in the form of stripes occurs in the picture signal. This phenomenon is called flicker. This results from the fact that a fluorescent lamp connected to a commercial power supply (AC) basically repeats flashing at a cycle twice the power supply frequency, and the operating principle of an imaging device.
Referring to FIG. 1, a description will be given of the principle of how flicker occurs in images shot with a CMOS image sensor. FIG. 1 shows the following pieces of data.
(a) Luminance variation of a fluorescent lamp
(b) Schematic diagram of the imaging (exposure) sequence of a CMOS image sensor
(c) Readout timing of CMOS images
(d) Conceptual illustration of output images
In FIG. 1, time (t) elapses from left to right.
It is assumed that the fluorescent lamp is operating in the 50 Hz commercial power supply region. In this case, the fluorescent lamp repeats flashing at a frequency of 100 Hz, which is twice the power supply frequency (50 Hz). The arcuate curves shown in (a) indicate luminance variation of the fluorescent lamp. Luminance variation is produced at 100 Hz, that is, in cycles of 1/100 seconds.
Under such an illumination environment, images are shot with a CMOS image sensor having a rolling shutter at a frame rate of 60 frames per second. The exposure process is performed sequentially from the top row (ROW) toward the bottom row (ROW) in each shot frame with the elapse of time.
In the schematic diagram of imaging of the CMOS image sensor shown in (b), the dotted diagonal lines are lines indicating reset timing of the image sensor, and solid diagonal lines are lines indicating readout timing. Exposure is started after the reset indicated by the dotted lines, and the period until the readout timing indicated by the solid lines is the exposure time. Exposure is performed from the scan line at the top of a frame toward the scan line at the bottom by the rolling shutter.
The region bounded by two adjacent solid diagonal lines indicates one frame of the image sensor. Within the exposure time sandwiched by each dotted line and solid line, luminance variation corresponding to luminance variation of illumination occurs. That is, since the exposure timing differs for each of rows that make up an image frame, due to the influence of the light source with luminance variation, non-uniformity in the form of horizontal stripes, or so-called flicker occurs as indicated by the (d) output images in FIG. 1.
The (d) output images in FIG. 1 indicate four consecutively shot images, frame #1 to frame #4, which are image frames #1 to #4 that make up moving images shot at a frame rate of 60 frames per second (60 fps).
A top portion p, a middle portion q, and a bottom portion r are shown in frame #3. These are shown for explicitly indicating positions corresponding to exposure times p, q, and r in the schematic diagram of imaging of the CMOS image sensor shown in (b).
The top portion p of frame #3 is the row portion subjected to exposure in a period during which the luminance of the fluorescent lamp is bright.
The middle portion q of frame #3 is the row portion subjected to exposure in a period during which the luminance of the fluorescent lamp changes from a dark period to a bright period.
The bottom portion r of frame #3 is the row portion subjected to exposure in a period during which the luminance of the fluorescent lamp is dark.
Stripe patterns based on luminance non-uniformity or color non-uniformity occurs because the fluorescent lamp luminance in the exposure period of each row (Row) is not consistent as described above.
It should be noted that a typical imaging device has such a configuration that, for example, one of RGB frequencies of light is selectively inputted for each of pixels that make up the imaging device. As this RGB arrangement, for example, the Bayer arrangement is known. For example, when imaging is performed with a color image sensor having the Bayer arrangement or the like, due to the decay characteristic of the phosphor of the fluorescent lamp, the degree of influence of flicker differs for each color signal (color channel), and color non-uniformity occurs due to the difference in amplitude and phase.
For example, PTL 1 (Japanese Unexamined Patent Application Publication No. 2007-174537) exists as related art disclosing a technology for preventing or suppressing such flicker. PTL 1 (Japanese Unexamined Patent Application Publication No. 2007-174537) discloses a process that controls the exposure time of the imaging device through setting of the electronic shutter or the like, thereby adjusting the exposure time of each row of a single shot image to reduce the difference in brightness between rows. However, there is a problem in that this technique places a constraint on the exposure time of the imaging device, making it impossible to set the electronic shutter in an arbitrary manner in accordance with the shooting environment, resulting in a decrease in the degree of freedom of shooting.
Also, to avoid the above-mentioned problem, PTL 2 (Japanese Unexamined Patent Application Publication No. 2005-347939) proposes a configuration that performs a correction process on an image signal obtained by a shooting process to thereby suppress the influence of fluorescent lamp flicker within the picture signal.
In this PTL 2 (Japanese Unexamined Patent Application Publication No. 2005-347939), the lighting waveform (flicker waveform) of a fluorescent lamp is approximately modeled by a sine wave, and then a correction process is performed by changing the correction gain used at the time of a correction process on an image signal shot with a camera, in accordance with the approximation model. Through this correction process, the image signal is corrected in accordance with the lighting waveform (flicker waveform) of the fluorescent lamp, thereby realizing a configuration that suppresses the difference in brightness between rows in the shot image.
However, the actual luminance variation of a fluorescent lamp does not necessarily match the sine wave as the approximation model used in PTL 2. FIG. 2 is a comparison diagram between the sinusoidal approximation model and an example of the actual luminance variation of a fluorescent lamp. For example, under such a condition that ripple is present in a commercial power supply, as shown in FIG. 2, the actual lighting waveform (solid line) of the fluorescent lamp and the model waveform (dotted line) exhibit different luminance variations.
The luminance variation of a typical fluorescent lamp does not coincide with the sinusoidal approximation model. In many cases, complex luminance variation is produced as indicated by the solid line shown in FIG. 2.
The process in PTL 2 described above does not take this divergence between the sinusoidal approximation model and reality into consideration, and thus there is a problem in that even when a flicker correction process using the model waveform is performed, the influence of flicker based on the real luminance variation of the fluorescent lamp remains in the image obtained as a result of the correction process.
The influence of flicker based on the real luminance variation of the fluorescent lamp becomes more pronounced as the exposure time becomes shorter. As the exposure becomes shorter, the interval between the solid and dotted diagonal lines shown in FIG. 1(b) becomes smaller, with a result that the influence of flicker based on the actual luminance variation of the fluorescent lamp becomes stronger.
FIG. 3 and FIG. 4 show variation of a flicker waveform due to the difference in exposure time of an imaging device (image sensor). It should be noted that herein, a flicker waveform is a waveform indicating luminance non-uniformity of each row which appears in the shot image.
Both FIG. 3 and FIG. 4 show the waveforms of flicker within the picture signal when shooting at 60 frames/second (60 fps) with a CMOS image sensor having a roller shutter. The flicker waveforms of frames #1 to #4 that are four consecutive frames are shown.
In each graph, the horizontal reading represents the row number in the image sensor, and the vertical axis represents the normalized luminance (flicker luminance) of each row. It should be noted that the normalized luminance represents information comparing relative luminance on a row-by-row basis which is generated while excluding the influence of the luminance of a subject.
FIG. 3 shows the flicker waveforms of frames #1 to #4 in the case when the exposure time of the image sensor is long at 1/60 [sec].
FIG. 4 shows the flicker waveforms of frames #1 to #4 in the case when the exposure time of the image sensor is short at 1/500 [sec].
In the case of the long exposure time of 1/60 [sec] shown in FIG. 3, the flicker waveform appearing in each of image frames #1 to #4 exhibits a gentle curve. This is a line close to the since wave as the model waveform described above with reference to FIG. 2.
On the other hand, in the case of the short exposure time of 1/500 [sec] shown in FIG. 4, unlike the curve close to the sine wave shown in FIG. 3, the flicker waveform appearing in each of image frames #1 to #4 is a line close to the actual waveform described above with reference to FIG. 2.
In this way, as the exposure time of the CMOS image sensor becomes shorter, the distribution of luminance non-uniformity/color non-uniformity in the form of horizontal stripes due to fluorescent lamp flicker which appears in the picture signal becomes closer to the actual flicker waveform.
In the case of high frame rate imaging which is becoming commonplace in recent years, and high speed shutter imaging in wide dynamic range imaging, shooting is executed with short exposure time. In such a shooting process, the divergence between the model waveform and the actual flicker waveform becomes apparent on the picture as well. Therefore, with the signal processing method using a model waveform described in PTL 2 mentioned above, it is not possible to perform effective correction that suppresses occurrence of flicker.